On the difficulty of training Recurrent Neural Networks

1 Analytical analysis of the exploding and vanishing gradients problem 1.1 Linear model x t = W rec σ(x t−1) + W in u t + b (1) Let us consider the term g T k = ∂Et ∂xt ∂xt ∂x k ∂ + x k ∂θ for the linear version of the parametriza-tion in equation (1) (i.e. set σ to the identity function) and assume t goes to infinity and l = t − k. We have that: ∂x t ∂x k = W T rec l (2) By employing a generic power iteration method based proof we can show that, given certain conditions, ∂Et ∂xt W T rec l grows exponentially.